Knowledge Distillation with Feature Self Attention

With the rapid development of deep learning technology, size and performance network continuously grow, making compression essential for commercial applications. In this paper, we propose a Feature Self Attention (FSA) module that extracts correlation information between hidden features new method distilling to compress model. FSA does not require special or match teacher model student By removing multi-head structure repeated self-attention blocks in existing mechanism, it minimizes addition parameters. Based on ResNet-18, 34, added parameters are only 2.00M training speed is also fastest comparison benchmark models. It was demonstrated through experiments use interrelationship loss can be beneficial models, indicating importance considering neural compression. And verified from scratch vanilla without pre-trained weight